Genetics of recent habitat contraction and reduction in population size: does isolation by distance matter?

Fragmentation and loss of natural habitats are recognized as major threats to contemporary flora and fauna. Detecting past or current reductions in population size is therefore a major aim in conservation genetics. Statistical methods developed to this purpose have tended to ignore the effects of spatial population structure. However in many species, individual dispersal is restricted in space and fine-scale spatial structure such as isolation by distance (IBD) is commonly observed in continuous populations. Using a simulation-based approach, we investigated how comparative and single-point methods, traditionally used in a Wright-Fisher (WF) population context for detecting population size reduction, behave for IBD populations. We found that a complex 'quartet' of factors was acting that includes restricted dispersal, population size (i.e. habitat size), demographic history, and sampling scale. After habitat reduction, IBD populations were characterized by a stronger inertia in the loss of genetic diversity than WF populations. This inertia increases with the strength of IBD, and decreases when the sampling scale increases. Depending on the method used to detect a population size reduction, a local sampling can be more informative than a sample scaled to habitat size or vice versa. However, IBD structure led in numerous cases to incorrect inferences on population demographic history. The reanalysis of a real microsatellite data set of skink populations from fragmented and intact rainforest habitats confirmed most of our simulation results.